Consumers have the need for producing photographic prints of different formats. Typically pictures captured on 35 mm film are of 2:3 aspect ratio. In order to produce prints of different aspect ratio, such as 3×5, 5×7, or 8×10 prints, an equal amount of cropping is applied to the two corresponding sides of the image (no cropping on the other two sides) in conventional print fulfillment. This procedure is called “crop to fill”. While in general this procedure produces satisfactory prints because photographers tend to frame the picture such that the main subject is in the middle of the picture, it also leads to catastrophic failures such as cropping off a subject's head when the subject's head is located near the top or a side border of the original image. The source of the problem is the lack of scene analysis to understand where the main subject and background are in the image.
Digital photography has created new opportunities as well as new problems. While many digital cameras produce digital images of 2:3 aspect ratio, a significant percentage of other cameras produce images of 3:4 aspect ratio. Because the most popular print format is 4×6, image cropping has to occur when printing digital images of an original aspect ratio of 3:4. A less than satisfactory alternative, called “crop to fit” as opposed to “crop to fill”, is to pad the 3:4 image with white space to obtain the desired aspect ratio.
Digital image processing enables a host of new possibilities. One such possibility is automatic scene-dependent image cropping, i.e., cropping undesirable content from a picture and magnifying or zooming the desired content to fill the entire photographic print. Bollman et al. in U.S. Pat. No. 5,978,519 describe a method for cropping images based upon the different intensity levels within the image. With this system, an image to be cropped is scaled down to a grid and divided into non-overlapping blocks. The mean and variance of intensity levels are calculated for each block. Based on the distribution of variances in the blocks, a threshold is selected for the variance. All blocks with a variance higher than the threshold variance are selected as regions of interest. The regions of interest are then cropped to a bounding rectangle. However, such a system is only effective when uncropped images contain regions where intensity levels are uniform and other regions where intensity levels vary considerably. In summary, this technique is only capable of cropping “open” space in the image and cannot deal with images with non-uniform background. Moreover, its cropping precision is also limited by the size of the non-overlapping blocks (i.e., sub-block cropping is not possible). As a result, there is no good way of enforcing a desired aspect ratio during the cropping. Another main drawback of this method is that it does not enforce any picture composition rules.
Another conventional method is employed by on-line print fulfillment service providers such as Ofoto. Assuming that all the pictures have been re-oriented if necessary to the upright orientation by a user upon preview, a somewhat “intelligent” cropping rule is to crop the image based on the so called “20–80” rule, i.e., apply 20% of the needed cropping amount at the top of the image and the remaining 80% at the bottom of the image to ensure that it is very unlikely that a subject's head would be cut off. This option is not possible in a film-based print fulfillment system because the assumption that the image is in the upright orientation is not always valid. In case cropping is needed for the left and right sides of the image (in upright orientation), an equal amount of cropping is applied to each side. Note that in either case, cropping is only applied to two sides of the original picture to maximally retain the content of the original image.
There is therefore a need to provide intelligent image cropping according to an automatic understanding of the image content and enforcement of compositional rules so that (1) the main subject of the image is not cropped in part or in its entirety, (2) both smooth and textured background can be identified and removed in part or in its entirety if necessary, and (3) common picture composition rules such as sufficient headroom can be enforced.